Perpendicular cut-in training

ABSTRACT

Techniques relating to training a model for detecting that a vehicle is likely to perform a cut-in maneuver are described. Computing device(s) can receive log data associated with vehicles in an environment and can detect an event in the log data, wherein an event corresponds to a cut-in maneuver performed by a vehicle. In an example, the computing device(s) can generate training data based at least in part on converting a portion of the log data that corresponds to the event into a top-down representation of the environment and inputting the training data into a model, wherein the model is trained to output an indication of whether another vehicle is likely to perform another cut-in maneuver.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a continuation application which claims priority to commonlyassigned, co-pending U.S. patent application Ser. No. 16/803,644, filedFeb. 27, 2020. Application Ser. No. 16/803,644 is fully incorporatedherein by reference.

BACKGROUND

In general, prediction systems utilize information associated withobjects in an environment to infer future actions of the objects, suchas trajectories. Such information can then be used to determine how tocontrol a vehicle, for example, in the environment.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears. Theuse of the same reference numbers in different figures indicates similaror identical components or features.

FIG. 1 illustrates an example environment for predicting whether avehicle is going to enter a lane region in front of another vehicle, asdescribed herein.

FIG. 2 illustrates another example environment for predicting whether avehicle is going to enter a lane region in front of another vehicle, asdescribed herein.

FIG. 3 illustrates another example environment for predicting whether avehicle is going to enter a lane region in front of another vehicle, asdescribed herein.

FIG. 4 illustrates another example environment for predicting whether avehicle is going to enter a lane region in front of another vehicle, asdescribed herein.

FIG. 5 illustrates another example environment for predicting whether avehicle is going to enter a lane region in front of another vehicle, asdescribed herein.

FIG. 6 is a block diagram illustrating an example system for performingtechniques, as described herein.

FIG. 7 illustrates an example process for training a model forpredicting whether a vehicle is going to enter a lane region in front ofanother vehicle, as described herein.

FIG. 8 illustrates an example process for predicting whether a vehicleis going to enter a lane region in front of another vehicle, andcontrolling the other vehicle based on such a prediction, as describedherein.

DETAILED DESCRIPTION

Techniques described herein are directed to training and using a modelto predict particular behaviors of objects (e.g., pedestrians, animals,cyclists, other vehicles, etc.) in an environment of a vehicle. Forinstance, a model can be trained using machine learning to predict whena vehicle that is proximate and substantially perpendicular to anothervehicle, such as an autonomous vehicle, is going to enter a lane regionin front of the autonomous vehicle (e.g., “cut-in”), such as by pullingout of a driveway, pulling out of a perpendicular parking spot, making aU-turn, or the like. In at least one example, the model can be trainedbased on training data previously collected by one or more vehicles inan environment. In at least one example, the model can be provided to anautonomous vehicle, and can be utilized by computing device(s) onboardthe autonomous vehicle to predict how objects (e.g., other vehicles)proximate and substantially perpendicular to the autonomous vehicle arelikely to behave.

For example, computing device(s) onboard the autonomous vehicle canleverage the model to predict when a vehicle that is proximate andsubstantially perpendicular to the autonomous vehicle is going to entera lane region in front of the autonomous vehicle (e.g., “cut-in”), suchas by pulling out of a driveway, pulling out of a perpendicular parkingspot, making a U-turn, or the like. These scenarios, in particular,present a unique challenge in that such agents (other vehiclesperforming these maneuvers) may begin from an unstructured portion of amap (e.g., in which there are no lane markers, such as in driveways orparking lots) in addition to the fact that agent motion may comprisetransitions to structured motions and/or complex motions frommulti-point turns which may not obey the rules of the road. Thecomputing device(s) onboard the autonomous vehicle can utilize suchpredictions to determine how to navigate the autonomous vehicle in theenvironment. That is, the computing device(s) onboard the autonomousvehicle can adapt driving operations based on predictions output by themodel. As such, the autonomous vehicle can utilize the model to (i)predict when a vehicle proximate and substantially perpendicular to theautonomous vehicle is going to cut-in and (ii) determine a trajectory tonavigate the autonomous vehicle to accommodate the other vehicle (e.g.,cause the autonomous vehicle to decelerate thereby increasing a followdistance between the vehicle and another vehicle, cause the autonomousvehicle to yield to the other vehicle and/or stop, cause the autonomousvehicle to perform a lane change operation, cause the autonomous vehicleto safely maneuver around another vehicle, and/or cause the autonomousvehicle to perform any other combination of maneuvers the autonomousvehicle is capable of performing).

Techniques described herein are directed to training a model, forexample, using machine learning, to enable a vehicle, such as anautonomous vehicle, to predict the behavior of objects in itsenvironment. Techniques described herein can utilize noisy sensor data,and more particularly, noisy prediction data (e.g., processed sensordata) to predict the behavior of objects in an environment associatedwith a vehicle more accurately than with existing prediction techniques.Furthermore, and as described above, with respect to vehicles that areperpendicular, or substantially perpendicular, to an autonomous vehicle,such vehicles may not be following a road network and/or following therules of the road for which the autonomous vehicle is trained toperceive and understand. As such, existing prediction techniques may beinsufficient for detecting and safely maneuvering around such vehicles.That is, techniques described herein provide a technological improvementover existing prediction technology. In addition to improving theaccuracy with which sensor data can be used to predict behaviors ofobjects in an environment of a vehicle, techniques described herein canreduce false positives and improve safety outcomes.

The techniques described herein can be implemented in a number of ways.Example implementations are provided below with reference to thefollowing figures. Example implementations are discussed in the contextof autonomous vehicles; however, the methods, apparatuses, andcomponents described herein can be applied to a variety of components(e.g., a sensor component or a robotic platform), and are not limited toautonomous vehicles. In one example, the techniques described herein maybe utilized in driver-controlled vehicles in which such a component mayprovide an indication to a driver of the vehicle of whether it is safeto perform various maneuvers. In another example, the techniques can beutilized in an aviation or nautical context, or in any componentinvolving objects or entities that may be associated with behavior thatis unknown to the component. In some examples, the techniques can beutilized in contexts outside of autonomous vehicles and/or perceptioncomponents. Furthermore, the techniques described herein can be usedwith real data (e.g., captured using sensor(s)), simulated data (e.g.,generated by a simulator), or any combination of the two.

FIG. 1 illustrates an example environment 100 for predicting whether avehicle is going to enter a lane region in front of another vehicle, asdescribed herein.

In the example environment 100, a vehicle 102 can be positioned in adriving lane 104 of a driving surface 106. In at least one example, thevehicle 102 can be an autonomous vehicle configured to operate accordingto a Level 5 classification issued by the U.S. National Highway TrafficSafety Administration, which describes a vehicle capable of performingall safety-critical functions for the entire trip, with the driver (oroccupant) not being expected to control the vehicle at any time. In suchan example, since the vehicle 102 can be configured to control allfunctions from start to stop, including all parking functions, it can beunoccupied. This is merely an example, and the components and methodsdescribed herein can be incorporated into any ground-borne, airborne, orwaterborne vehicle, including those ranging from vehicles that need tobe manually controlled by a driver at all times, to those that arepartially or fully autonomously controlled. That is, in the illustratedexample, the vehicle 102 is an autonomous vehicle; however, the vehicle102 could be any other type of vehicle.

In at least one example, the driving lane 104 can be associated with alane region and the driving surface 106 can be associated with a road.In at least one example, another vehicle 108 can be detected within theenvironment 100. In some examples, the other vehicle 108 can beproximate to the vehicle 102 such that the other vehicle 108 is within athreshold distance of the vehicle 102. In some examples, the othervehicle 108 can be associated with a direction of travel that isdifferent from the vehicle 102. The direction of travel of the vehicle102 is illustrated by the arrow at the end of the trajectory 110 alongwhich the vehicle 102 is travelling. In some examples, the other vehicle108 can be associated with a direction of travel that is opposite thedirection of travel of the vehicle 102. In other examples, a directionof travel can be different from another direction of travel by beingoffset from the other direction of travel by more than a threshold.

In some examples, the other vehicle 108 can be positioned at an angle tothe vehicle 102. In at least one example, the angle can be within adesignated offset of 90 degrees such that the other vehicle 108 issubstantially perpendicular to the vehicle 102. That is, for the purposeof this discussion, a vehicle is “substantially perpendicular” toanother vehicle if an angle between the two vehicles is within adesignated offset of 90 degrees. As a non-limiting example, thedesignated offset can be 50 degrees such that the vehicle 102 and theother vehicle 108 (or any two vehicles) can be “substantiallyperpendicular” so long as the angle between the two is between 40degrees and 140 degrees. In at least one example, the designated offsetcan be learned using machine learning techniques described herein. Insome examples, the designated offset can be set manually, can be basedon map data (e.g., may be dependent on location), and the like.

In FIG. 1 , the other vehicle 108 is positioned substantiallyperpendicular to the vehicle 102. As a non-limiting example, the othervehicle 108 can be pulling out of a parking spot (e.g., a substantiallyperpendicular parking spot) or parking lot, an alley, a driveway, or anyother location that is substantially perpendicular to the vehicle 102.In some examples, such a parking spot or parking lot, alley, driveway,etc. can be annotated in a map associated with the environment 100. Inother examples, such a parking spot or parking lot, alley, driveway,etc. may not be annotated in the map.

In at least one example, the vehicle 102 can include one or morecomputing devices, which can be onboard the vehicle 102. In at least oneexample, the computing device(s) can include components for controllingthe vehicle 102. For instance, in at least one example, a perceptioncomponent can perform object detection, segmentation, and/orclassification based at least in part on sensor data received fromsensor component(s) of the vehicle 102. In at least one example, aprediction component can receive sensor data from the sensorcomponent(s), map data associated with a map (e.g., of the environment100), and/or perception data output from the perception component (e.g.,processed sensor data), and can output predictions associated with oneor more objects within the environment 100 of the vehicle 102. In atleast one example, a planning component can determine outputs to use tocontrol the vehicle 102 based at least in part on sensor data receivedfrom the sensor component(s), map data, and/or any determinations madeby the other components of the vehicle 102. Additional detailsassociated with such computing device(s) and associated components aredescribed below with reference to FIG. 6 .

As described above, the vehicle 102 can be associated with one or moresensor components. In at least one example, the sensor component(s) caninclude lidar sensors, radar sensors, ultrasonic transducers, sonarsensors, location sensors (e.g., global positioning component (GPS),compass, etc.), inertial sensors (e.g., inertial measurement units,accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB,IR, intensity, depth, etc.), wheel encoders, microphones, environmentsensors (e.g., temperature sensors, humidity sensors, light sensors,pressure sensors, etc.), time of flight (ToF) sensors, etc. Such sensordata can include, but is not limited to, lidar data, radar data,ultrasonic transducer data, sonar data, location data (e.g., globalpositioning component (GPS), compass, etc.), inertial data (e.g.,inertial measurement units data, accelerometer data, magnetometer data,gyroscope data, etc.), camera data (e.g., RGB data, IR data, intensitydata, depth data, etc.), wheel encoder data, microphone data,environment sensor data (e.g., temperature sensor data, humidity sensordata, light sensor data, pressure sensor data, etc.), ToF sensor data,etc.

In at least one example, the perception component can detect the othervehicle 108 and the prediction component can predict whether the othervehicle 108 is going to enter the lane region of the driving lane 104 infront of the vehicle 102. In some examples, the prediction component canleverage a model, trained by machine learning mechanism(s), to output aprediction indicating whether the other vehicle 108 is going to enterthe lane region of the driving lane 104 in front of the vehicle 102. Inat least one example, the prediction component can analyze feature(s)(also referred to as attributes, characteristics, parameters, data, andthe like) associated with the environment 100 to determine whether theother vehicle 108 is likely to enter the lane region of the driving lane104. Such feature(s), which can be associated with characteristic(s) ofthe environment 100, can include annotations in a map associated withthe environment 100 indicating that the other vehicle 108 is located ina parking spot (e.g., a substantially perpendicular parking spot) orparking lot, an alley, a driveway, etc.

In at least one example, the prediction component can analyze feature(s)associated with the other vehicle 108 to determine whether the othervehicle 108 is likely to enter the lane region of the driving lane 104.For example, such feature(s), which can be associated withcharacteristic(s) of the other vehicle 108, can include, but are notlimited to, a position of the other vehicle 108 relative to the vehicle102 (e.g., direction of travel, angular offset, etc.), an instantaneousvelocity of the other vehicle 108, an indication of whether a driver isin the other vehicle 108, an indication of a direction the driver islooking (e.g., head position of the driver), an indication of whether adoor associated with the other vehicle 108 is open or closed, anindication of whether an engine of the other vehicle 108 is in a runningstate or off state, a wheel angle associated with wheel(s) of the othervehicle 108 (additional details for determining such are described inU.S. patent application Ser. No. 16/709,263, which was filed on Dec. 10,2019, and is incorporated by reference herein in its entirety), anindication of whether a brake light of the other vehicle 108 isilluminated, an indication of whether a headlight of the other vehicle108 is illuminated, an indication of whether a reverse light of theother vehicle 108 is illuminated, an indication of whether a blinker ofthe other vehicle 108 is illuminated, or a lighting state associatedwith the other vehicle 108.

Additional details associated with such a model, and techniques fortraining such a model, are described below with reference to FIG. 7 .

As illustrated in FIG. 1 , if the other vehicle 108 follows thetrajectory 112 associated therewith, the other vehicle 108 will enterthe lane region of the driving lane 104 in front of the vehicle 102(e.g., as it performs a right turn into traffic). That is, if the othervehicle 108 follows the trajectory 112 associated therewith, the othervehicle 108 will perform a cut-in, such that the planning componentassociated with the vehicle 102 may modify its trajectory 110 tomaneuver the vehicle 102 in view of the other vehicle 202. As describedabove, the planning component can cause the vehicle 102 to deceleratesuch to increase a follow distance between the vehicle 102 and the othervehicle 108, cause the vehicle 102 to decelerate and yield to the othervehicle 108 and/or stop, cause the vehicle 102 to perform a lane change,cause the vehicle 102 to safely maneuver around the other vehicle 108,and/or cause the vehicle 102 to perform any other combination ofmaneuvers that the vehicle 102 is capable of performing.

FIG. 2 illustrates another example environment 200 for predictingwhether a vehicle is going to enter a lane region in front of anothervehicle, as described herein.

The example environment 200 includes the vehicle 102, which can bepositioned in the driving lane 104 of the driving surface 106. In atleast one example, the driving lane 104 can be associated with a laneregion and the driving surface 106 can be associated with a road, asdescribed above with reference to FIG. 1 . In at least one example,another vehicle 202 can be detected within the environment 200. In someexamples, the other vehicle 202 can be proximate to the vehicle 102 suchthat the other vehicle 202 is within a threshold distance of the vehicle102. Further, in some examples, the other vehicle 202 can be associatedwith a direction of travel that is different from the vehicle 102. Thedirection of travel of the vehicle 102 is illustrated by the arrow atthe end of the trajectory 110 along which the vehicle 102 is travelling.In some examples, the other vehicle 202 can be associated with adirection of travel that is opposite the direction of travel of thevehicle 102. In some examples, the other vehicle 202 can be positionedat an angle to the vehicle 102. In at least one example, the angle canbe within a designated offset of 90 degrees such that the other vehicle202 is substantially perpendicular to the vehicle 102.

In FIG. 2 , the other vehicle 202 is positioned substantiallyperpendicular to the vehicle 102. As a non-limiting example, the othervehicle 108 can be pulling out of a parking spot (e.g., a substantiallyperpendicular parking spot) or parking lot, an alley, a driveway, or anyother location that is substantially perpendicular to the vehicle 102.In some examples, such a parking spot or parking lot, alley, driveway,etc. can be annotated in a map associated with the environment 200. Inother examples, such a parking spot or parking lot, alley, driveway,etc. may not be annotated in the map.

In at least one example, the perception component can detect the othervehicle 202 and the prediction component can output a predictionindicating whether the other vehicle 202 is going to enter the laneregion of the driving lane 104 in front of the vehicle 102. In someexamples, the prediction component can leverage a model, trained bymachine learning mechanism(s), to predict whether the other vehicle 202is going to enter the lane region of the driving lane 104 in front ofthe vehicle 102. In at least one example, the prediction component cananalyze feature(s) associated with the environment 200 and/or the othervehicle 202 to determine whether the other vehicle 202 is likely toenter the lane region of the driving lane 104, as described above.

As illustrated in FIG. 2 , if the other vehicle 202 follows thetrajectory 204 associated therewith, the other vehicle 202 will enterthe lane region of the driving lane 104 in front of the vehicle 102(e.g., as it performs a left turn into traffic). That is, if the othervehicle 202 follows the trajectory 204 associated therewith, the othervehicle 202 will perform a cut-in, such that the planning componentassociated with the vehicle 102 may modify its trajectory 110 tomaneuver the vehicle 102 in view of the other vehicle 202. As describedabove, the planning component can cause the vehicle 102 to deceleratesuch to increase a follow distance between the vehicle 102 and the othervehicle 202, cause the vehicle 102 to decelerate to yield to the othervehicle 202 and/or stop, and/or cause the vehicle 102 to perform anyother combination of maneuvers the vehicle 102 is capable of performing.

FIGS. 1 and 2 illustrate examples of techniques described herein wherebythe other vehicle 108 and 202, respectively, are pulling out of analley, parking spot (e.g., substantially perpendicular parking spot) orparking lot, driveway, or other location that is substantiallyperpendicular to the vehicle 102 and/or the driving surface 106. In FIG.1 , the other vehicle 108 is positioned in an alley, parking spot (e.g.,substantially perpendicular parking spot) or parking lot, driveway, orother location that is on a same side of the driving surface 106 as thevehicle 102. That is, the other vehicle 108 is positioned substantiallyperpendicular to, and on a same side of the driving surface 106 as, thedriving lane 104 in which the vehicle 102 is positioned. In FIG. 2 , theother vehicle 202 is positioned in an alley, parking spot (e.g.,substantially perpendicular parking spot) or parking lot, driveway, orother location that is on an opposite side of the driving surface 106 asthe vehicle 102. That is, the other vehicle 108 is positionedsubstantially perpendicular to, and on an opposite side of the drivingsurface 106 as, the driving lane 104 in which the vehicle 102 ispositioned.

In some examples, the other vehicle 108 or 202 can be positioned infront of the vehicle 102 and positioned substantially parallel (e.g., ina same direction as) the vehicle 102. In such examples, the othervehicle 108 or 202 can perform a left or right turn across the drivinglane 104. The vehicle 102 can leverage feature(s) as described above topredict such a maneuver and the planning component can determine how tocontrol the vehicle 102 accordingly.

FIG. 3 illustrates another example environment 300 for predictingwhether a vehicle is going to enter a lane region in front of anothervehicle, as described herein.

The example environment 300 includes the vehicle 102, which can bepositioned in the driving lane 104 of the driving surface 106. In atleast one example, the driving lane 104 can be associated with a laneregion and the driving surface 106 can be associated with a road, asdescribed above with reference to FIG. 1 . In at least one example,another vehicle 302 can be detected within the environment 300. In someexamples, the other vehicle 302 can be proximate to the vehicle 102 suchthat the other vehicle 302 is within a threshold distance of the vehicle102. Further, in some examples, the other vehicle 302 can be associatedwith a direction of travel that is different from the vehicle 102. Thedirection of travel of the vehicle 102 is illustrated by the arrow atthe end of the trajectory 110 along which the vehicle 102 is travelling.In some examples, the other vehicle 302 can be associated with adirection of travel that is opposite the direction of travel of thevehicle 102 (e.g., as illustrated in FIG. 3 ). In some examples, theother vehicle 302 can be positioned at an angle to the vehicle 102.

In at least one example, the perception component can detect the othervehicle 302 and the prediction component can predict whether the othervehicle 302 is going to enter the lane region of the driving lane 104 infront of the vehicle 102. In some examples, the prediction component canleverage a model, trained by machine learning mechanism(s), to output aprediction indicating whether the other vehicle 302 is going to enterthe lane region of the driving lane 104 in front of the vehicle 102. Inat least one example, the prediction component can analyze feature(s)associated with the environment 300 and/or the other vehicle 302 todetermine whether the other vehicle 302 is likely to enter the laneregion of the driving lane 104, as described above.

As illustrated in FIG. 3 , if the other vehicle 302 follows thetrajectory 304 associated therewith, the other vehicle 302 will enterthe lane region of the driving lane 104 in front of the vehicle 102(e.g., as it performs a U-turn). That is, if the other vehicle 302follows the trajectory 304 associated therewith, the other vehicle 302will perform a cut-in, such that the planning component associated withthe vehicle 102 may modify its trajectory 110 to maneuver the vehicle102 in view of the other vehicle 302. As described above, the planningcomponent can cause the vehicle 102 to decelerate such to increase afollow distance between the vehicle 102 and the other vehicle 302, causethe vehicle 102 to decelerate to yield to the other vehicle 302 and/orstop, and/or cause the vehicle 102 to perform any other combination ofmaneuvers the vehicle 102 is capable of performing.

While FIG. 3 illustrates the other vehicle 302 performing a u-turn, inan alternative example, the other vehicle 302 may perform a turn acrossthe driving lane 104 (e.g., into a driveway, street, alley, or thelike). In such an example, the vehicle 102 can leverage the samefeature(s) to determine whether the other vehicle 302 is likely to enterthe lane region of the driving lane 104 in front of the vehicle 102 andthe planning component can determine how to control the vehicle 102accordingly.

FIG. 4 illustrates another example environment 400 for predictingwhether a vehicle is going to enter a lane region in front of anothervehicle, as described herein.

The example environment 400 includes the vehicle 102, which can bepositioned in the driving lane 104 of the driving surface 106. In atleast one example, the driving lane 104 can be associated with a laneregion and the driving surface 106 can be associated with a road, asdescribed above with reference to FIG. 1 . In at least one example,another vehicle 402 can be detected within the environment 400. In someexamples, the other vehicle 402 can be proximate to the vehicle 102 suchthat the other vehicle 402 is within a threshold distance of the vehicle102. Further, in some examples, the other vehicle 402 can be associatedwith a direction of travel that is different from the vehicle 102. Thedirection of travel of the vehicle 102 is illustrated by the arrow atthe end of the trajectory 110 along which the vehicle 102 is travelling.In some examples, the other vehicle 402 can be associated with adirection of travel that is opposite the direction of travel of thevehicle 102 (e.g., as illustrated in FIG. 4 ). In some examples, theother vehicle 402 can be positioned at an angle to the vehicle 102.

In at least one example, the perception component can detect the othervehicle 402 and the prediction component can predict whether the othervehicle 402 is going to enter the lane region of the driving lane 104 infront of the vehicle 102. In some examples, the prediction component canleverage a model, trained by machine learning mechanism(s), to output aprediction indicating whether the other vehicle 402 is going to enterthe lane region of the driving lane 104 in front of the vehicle 102. Inat least one example, the prediction component can analyze feature(s)associated with the environment 400 and/or the other vehicle 402 todetermine whether the other vehicle 402 is likely to enter the laneregion of the driving lane 104, as described above.

As illustrated in FIG. 4 , if the other vehicle 402 follows thetrajectory 404 associated therewith, the other vehicle 402 will enterthe lane region of the driving lane 104 in front of the vehicle 102(e.g., as it performs a three-point turn, or any N-point turn). That is,if the other vehicle 402 follows the trajectory 404 associatedtherewith, the other vehicle 402 will perform a cut-in, such that theplanning component associated with the vehicle 102 may modify itstrajectory 110 to maneuver the vehicle 102 in view of the other vehicle402. As described above, the planning component can cause the vehicle102 to decelerate such to increase a follow distance between the vehicle102 and the other vehicle 402, cause the vehicle 102 to decelerate toyield to the other vehicle 402 and/or stop, and/or cause the vehicle 102to perform any other combination of maneuvers the vehicle 102 is capableof performing.

FIG. 5 illustrates another example environment 500 for predictingwhether a vehicle is going to enter a lane region in front of anothervehicle, as described herein.

The example environment 500 includes the vehicle 102, which can bepositioned in the driving lane 104 of the driving surface 106. In atleast one example, the driving lane 104 can be associated with a laneregion and the driving surface 106 can be associated with a road, asdescribed above with reference to FIG. 1 . In at least one example,another vehicle 502 can be detected within the environment 500. In someexamples, the other vehicle 502 can be proximate to the vehicle 102 suchthat the other vehicle 502 is within a threshold distance of the vehicle102. Further, in some examples, the other vehicle 502 can be associatedwith a direction of travel that is different from the vehicle 102. Thedirection of travel of the vehicle 102 is illustrated by the arrow atthe end of the trajectory 110 along which the vehicle 102 is travelling.In some examples, the other vehicle 502 can be associated with adirection of travel that is opposite the direction of travel of thevehicle 102. In some examples, the other vehicle 502 can be positionedat an angle to the vehicle 102. As illustrated in FIG. 5 , the othervehicle 502 can be substantially perpendicular to the vehicle 102.

In at least one example, the perception component can detect the othervehicle 502 and the prediction component can predict whether the othervehicle 502 is going to enter the lane region of the driving lane 104 infront of the vehicle 102. In some examples, the prediction component canleverage a model, trained by machine learning mechanism(s), to output aprediction indicating whether the other vehicle 502 is going to enterthe lane region of the driving lane 104 in front of the vehicle 102. Inat least one example, the prediction component can analyze feature(s)associated with the environment 500 and/or the other vehicle 502 todetermine whether the other vehicle 502 is likely to enter the laneregion of the driving lane 104, as described above.

As illustrated in FIG. 5 , if the other vehicle 502 follows thetrajectory 504 associated therewith, the other vehicle 502 will enterthe lane region of the driving lane 104 in front of the vehicle 102(e.g., as it reverses from its current position and proceeds in the samedirection of travel as the vehicle 102). That is, if the other vehicle502 follows the trajectory 504 associated therewith, the other vehicle502 will perform a cut-in, such that the planning component associatedwith the vehicle 102 may modify its trajectory 110 to maneuver thevehicle 102 in view of the other vehicle 502. As described above, theplanning component can cause the vehicle 102 to decelerate such toincrease a follow distance between the vehicle 102 and the other vehicle502, cause the vehicle 102 to change lanes, cause the vehicle 102 todecelerate to yield to the other vehicle 502 and/or stop, cause thevehicle 102 to safely maneuver around the other vehicle 502, and/orcause the vehicle 102 to perform any other combination of maneuvers thevehicle 102 is capable of performing.

While FIG. 5 illustrates the other vehicle 502 reversing out of adriveway, parking spot, alley, or the like on a same side of the drivingsurface 106 as the vehicle 102, techniques described herein are alsoapplicable for another vehicle 502 that is reversing out of a driveway,parking spot, alley, or the like on an opposite side of the drivingsurface 106 as the vehicle 102.

FIGS. 1-5 illustrate various examples of other vehicles that areproximate to the vehicle 102 that are entering a lane region of thedriving lane 104 in front of the vehicle 102 or are otherwise performinga “cut-in maneuver,” as described herein. Such examples include leftturns, right turns, U-turns, three-point turns, and the like. Additionalor alternative examples are within the scope of this disclosure. Forinstance, techniques described herein can be used for determining how tocontrol the vehicle 102 when another vehicle performs a K-turn, anyN-point turn, or the like, or another vehicle turning into or out oftraffic. That is, techniques described herein are not limited to theexamples described in FIGS. 1-5 and can be applicable to any examplewhere another vehicle that is proximate to and associated with a same ordifferent direction of travel than the vehicle 102 enters—or ispredicted to enter—a lane region of the driving lane 104 in front of thevehicle 102. Furthermore, techniques described herein are applicable toscenarios where another vehicle is entering a lane region from aposition on-road (e.g., on the driving surface 106) or off-road (e.g.,not on the driving surface 106).

FIG. 6 is a block diagram illustrating an example system 600 forperforming techniques, as described herein. In at least one example, avehicle 602 can include one or more vehicle computing devices 604, oneor more sensor components 606, one or more emitters 608, one or morecommunication connections 610, at least one direct connection 612, andone or more drive systems 614. In at least one example, a vehicle 602can correspond to the vehicle 102 described above with reference to FIG.1 . As described above with reference to FIG. 1 , in the illustratedexample, the vehicle 602 is an autonomous vehicle; however, the vehicle602 could be any other type of vehicle. While only a single vehicle 602is illustrated in FIG. 6 , in a practical application, the examplesystem 600 can include a plurality of vehicles, which, in some examples,can comprise a fleet of vehicles.

The vehicle computing device(s) 604 can include processor(s) 616 andmemory 618 communicatively coupled with the processor(s) 616. In theillustrated example, the memory 618 of the vehicle computing device(s)604 stores a localization component 620, a perception component 622, aprediction component 624, a planning component 626, and one or moresystem controllers 628. Additionally, the memory 618 can include astorage 630, which can store map(s), model(s), previous outputs, etc. Amap can be any number of data structures that are capable of providinginformation about an environment, such as, but not limited to,topologies (such as junctions, lanes, merging zones, etc.), streets,mountain ranges, roads, terrain, and the environment in general. Mapscan be associated with real environments or simulated environments.Model(s) can include machine-trained models, as described below. In someexamples, the storage 630 can store previous outputs.

In at least one example, the localization component 620 can determine apose (position and orientation) of the vehicle 602 in relation to alocal and/or global map based at least in part on sensor data receivedfrom the sensor component(s) 606 and/or map data associated with a map(e.g., of the map(s)). In at least one example, the localizationcomponent 620 can include, or be associated with, a calibrationcomponent that is capable of performing operations for calibrating(determining various intrinsic and extrinsic parameters associated withany one or more of the sensor component(s) 606), localizing, and mappingsubstantially simultaneously.

In at least one example, the perception component 622 can perform objectdetection, segmentation, and/or classification based at least in part onsensor data received from the sensor component(s) 606. In at least oneexample, the perception component 622 can receive raw sensor data (e.g.,from the sensor component(s) 606). In at least one example, theperception component 622 can receive sensor data and can utilize one ormore processing algorithms to perform object detection, segmentation,and/or classification with respect to object(s) identified in the sensordata. In some examples, the perception component 622 can associate abounding region (or otherwise an instance segmentation) with anidentified object and can associate a confidence score associated with aclassification of the identified object with the identified object. Insome examples, objects, when rendered via a display, can be coloredbased on their perceived class.

The prediction component 624 can receive sensor data from the sensorcomponent(s) 606, map data associated with a map (e.g., of the map(s)which can be in storage 630), and/or perception data output from theperception component 622 (e.g., processed sensor data), and can outputpredictions associated with one or more objects within the environmentof the vehicle 602. In at least one example, the planning component 626can determine outputs, to use to control the vehicle 602 based at leastin part on sensor data received from the sensor component(s) 606, mapdata, and/or any determinations made by the other components of thevehicle 602.

Additional details of localization components, perception components,prediction components, and/or planning components that are usable can befound in U.S. Pat. No. 9,612,123, issued on Apr. 4, 2017, and U.S. Pat.No. 10,353,390, issued on Jul. 16, 2019, the entire contents of both ofwhich are incorporated by reference herein. In some examples (e.g.,where the vehicle 602 is not an autonomous vehicle), one or more of theaforementioned components can be omitted from the vehicle 602. While thecomponents described above are illustrated as “onboard” the vehicle 602,in other implementations, the components can be remotely located and/oraccessible to the vehicle 602. Furthermore, while the components aredescribed above as “components,” such components can comprise one ormore components, which can be part of a system, for performingoperations attributed to each of the components.

In at least one example, the localization component 620, the perceptioncomponent 622, the prediction component 624, and/or the planningcomponent 626 can process sensor data, as described above, and can sendtheir respective outputs over network(s) 632, to computing device(s)634. In at least one example, the localization component 620, theperception component 622, the prediction component 624, and/or theplanning component 626 can send their respective outputs to thecomputing device(s) 634 at a particular frequency, after a lapse of apredetermined period of time, in near real-time, etc.

In at least one example, the vehicle computing device(s) 604 can includeone or more system controllers 628, which can be configured to controlsteering, propulsion, braking, safety, emitters, communication, andother systems of the vehicle 602. These system controller(s) 628 cancommunicate with and/or control corresponding systems of the drivesystem(s) 614 and/or other systems of the vehicle 602.

In at least one example, the sensor component(s) 606 can include lidarsensors, radar sensors, ultrasonic transducers, sonar sensors, locationsensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertialmeasurement units, accelerometers, magnetometers, gyroscopes, etc.),cameras (e.g., RGB, IR, intensity, depth, etc.), wheel encoders, audiosensors, environment sensors (e.g., temperature sensors, humiditysensors, light sensors, pressure sensors, etc.), ToF sensors, etc. Thesensor component(s) 606 can provide input to the vehicle computingdevice(s) 604. In some examples, the sensor component(s) 606 canpreprocess at least some of the sensor data prior to sending the sensordata to the vehicle computing device(s) 604. In at least one example,the sensor component(s) 606 can send sensor data, via the network(s)632, to the computing device(s) 634 at a particular frequency, after alapse of a predetermined period of time, in near real-time, etc.

The vehicle 602 can also include one or more emitters 608 for emittinglight and/or sound, as described above. The emitter(s) 608 in thisexample include interior audio and visual emitters to communicate withpassengers of the vehicle 602. By way of example and not limitation,interior emitters can include speakers, lights, signs, display screens,touch screens, haptic emitters (e.g., vibration and/or force feedback),mechanical actuators (e.g., seatbelt tensioners, seat positioners,headrest positioners, etc.), and the like. The emitter(s) 608 in thisexample also include exterior emitters. By way of example and notlimitation, the exterior emitters in this example include light emitters(e.g., indicator lights, signs, light arrays, etc.) to visuallycommunicate with pedestrians, other drivers, other nearby vehicles,etc., one or more audio emitters (e.g., speakers, speaker arrays, horns,etc.) to audibly communicate with pedestrians, other drivers, othernearby vehicles, etc., etc. In at least one example, the emitter(s) 608can be positioned at various locations about the exterior and/orinterior of the vehicle 602.

The vehicle 602 can also include communication connection(s) 610 thatenable communication between the vehicle 602 and other local or remotecomputing device(s). For instance, the communication connection(s) 610can facilitate communication with other local computing device(s) on thevehicle 602 and/or the drive system(s) 614. Also, the communicationconnection(s) 610 can allow the vehicle to communicate with other nearbycomputing device(s) (e.g., other nearby vehicles, traffic signals,etc.). The communications connection(s) 610 also enable the vehicle 602to communicate with a remote teleoperations computing device or otherremote services.

The communications connection(s) 610 can include physical and/or logicalinterfaces for connecting the vehicle computing device(s) 604 to anothercomputing device or a network, such as network(s) 632. For example, thecommunications connection(s) 610 can enable Wi-Fi-based communicationsuch as via frequencies defined by the IEEE 802.11 standards, shortrange wireless frequencies such as BLUETOOTH®, or any suitable wired orwireless communications protocol that enables the respective computingdevice to interface with the other computing device(s).

The direct connection 612 can directly connect the drive system(s) 614and other systems of the vehicle 602.

In at least one example, the vehicle 602 can include drive system(s)614. In some examples, the vehicle 602 can have a single drive system614. In at least one example, if the vehicle 602 has multiple drivesystems 614, individual drive systems 614 can be positioned on oppositeends of the vehicle 602 (e.g., the front and the rear, etc.). In atleast one example, the drive system(s) 614 can include sensorcomponent(s) to detect conditions of the drive system(s) 614 and/or thesurroundings of the vehicle 602. By way of example and not limitation,the sensor component(s) can include wheel encoder(s) (e.g., rotaryencoders) to sense rotation of the wheels of the drive system, inertialsensors (e.g., inertial measurement units, accelerometers, gyroscopes,magnetometers, etc.) to measure position and acceleration of the drivesystem, cameras or other image sensors, ultrasonic sensors toacoustically detect objects in the surroundings of the drive system,lidar sensors, radar sensors, etc. Some sensors, such as the wheelencoder(s), can be unique to the drive system(s) 614. In some cases, thesensor component(s) on the drive system(s) 614 can overlap or supplementcorresponding systems of the vehicle 602 (e.g., sensor component(s)606).

The drive system(s) 614 can include many of the vehicle systems,including a high voltage battery, a motor to propel the vehicle 602, aninverter to convert direct current from the battery into alternatingcurrent for use by other vehicle systems, a steering system including asteering motor and steering rack (which can be electric), a brakingsystem including hydraulic or electric actuators, a suspension systemincluding hydraulic and/or pneumatic components, a stability controlsystem for distributing brake forces to mitigate loss of traction andmaintain control, an HVAC system, lighting (e.g., lighting such ashead/tail lights to illuminate an exterior surrounding of the vehicle),and one or more other systems (e.g., cooling system, safety systems,onboard charging system, other electrical components such as a DC/DCconverter, a high voltage junction, a high voltage cable, chargingsystem, charge port, etc.). Additionally, the drive system(s) 614 caninclude a drive system controller which can receive and preprocess datafrom the sensor component(s) and to control operation of the variousvehicle systems. In some examples, the drive system controller caninclude processor(s) and memory communicatively coupled with theprocessor(s). The memory can store one or more components to performvarious functionalities of the drive system(s) 614. Furthermore, thedrive system(s) 614 also include communication connection(s) that enablecommunication by the respective drive system with other local or remotecomputing device(s).

In FIG. 6 , the vehicle computing device(s) 604, sensor component(s)606, emitter(s) 608, and the communication connection(s) 610 are shownonboard the vehicle 602. However, in some examples, the vehiclecomputing device(s) 604, sensor component(s) 606, emitter(s) 608, andthe communication connection(s) 610 can be implemented outside of anactual vehicle (i.e., not onboard the vehicle 602).

As described above, the vehicle 602 can send sensor data to thecomputing device(s) 634, via the network(s) 632. In some examples, thevehicle 602 can send raw sensor data to the computing device(s) 634. Inother examples, the vehicle 602 can send processed sensor data and/orrepresentations of sensor data to the computing device(s) 634 (e.g.,data output from the localization component 620, the perceptioncomponent 622, the prediction component 624, and/or the planningcomponent 626). In some examples, the vehicle 602 can send sensor datato the computing device(s) 634 at a particular frequency, after a lapseof a predetermined period of time, in near real-time, etc.

The computing device(s) 634 can receive the sensor data (raw orprocessed) from the vehicle 602 and/or other data collection devices, aswell as data from one or more third party sources and/or systems. In atleast one example, the computing device(s) 634 can include processor(s)636 and memory 638 communicatively coupled with the processor(s) 636. Inthe illustrated example, the memory 638 of the computing device(s) 634stores a training component 640, which can include an annotationcomponent 642, a map(s) storage 644 (e.g., storing one or more maps), atraining data storage 646 (e.g., storing training data accessible to thetraining component 640), and a model(s) storage 648 (e.g., models outputby the training component 640). In some examples, one or more of thesystems and/or storage repositories can be associated with the vehicle602 or other computing device(s) associated with the system 600 insteadof, or in addition to, being associated with the memory 638 of thecomputing device(s) 634.

In at least one example, the training component 640 can train model(s),which can be used for various operations as described herein. That is,the training component 640 can train model(s) that can be used forpredicting whether a first vehicle is going to enter a driving lane infront of another vehicle (e.g., the vehicle 602). In at least oneexample, the annotation component 642 can receive or otherwise determineannotated data based at least in part on sensor data and/or log dataassociated with the sensor data received from one or more vehicles, asdescribed below. Details associated with training the model(s), as usedherein, are described below with reference to FIG. 7 . In at least oneexample, the resulting model(s) can be stored in the model(s) storage648 and/or the storage 630 on the vehicle 602 and can be accessed innear real-time by one or more components of the vehicle computingdevice(s) 604.

The processor(s) 616 of the vehicle 602 and the processor(s) 636 of thecomputing device(s) 634 can be any suitable processor capable ofexecuting instructions to process data and perform operations asdescribed herein. By way of example and not limitation, the processor(s)616 and 636 can comprise one or more Central Processing Units (CPUs),Graphics Processing Units (GPUs), or any other device or portion of adevice that processes electronic data to transform that electronic datainto other electronic data that can be stored in registers and/ormemory. In some examples, integrated circuits (e.g., ASICs, etc.), gatearrays (e.g., FPGAs, etc.), and other hardware devices can also beconsidered processors in so far as they are configured to implementencoded instructions.

Memory 618 and 638 are examples of non-transitory computer-readablemedia. Memory 618 and 638 can store an operating system and one or moresoftware applications, instructions, programs, and/or data to implementthe methods described herein and the functions attributed to the varioussystems. In various implementations, the memory can be implemented usingany suitable memory technology, such as static random receive memory(SRAM), synchronous dynamic RAM (SDRAM), nonvolatile/Flash-type memory,or any other type of memory capable of storing information. Thearchitectures, systems, and individual elements described herein caninclude many other logical, programmatic, and physical components, ofwhich those shown in the accompanying figures are merely examples thatare related to the discussion herein.

It should be noted that while FIG. 6 is illustrated as a distributedsystem, in some examples, components of the vehicle 602 can beassociated with the computing device(s) 634 and/or the components of thecomputing device(s) 634 can be associated with the vehicle 602. That is,the vehicle 602 can perform one or more of the functions associated withthe computing device(s) 634, and vice versa.

FIGS. 7 and 8 are flowcharts showing example processes involvingtechniques as described herein. The processes illustrated in FIGS. 7 and8 are described with reference to the system 600 shown in FIG. 6 forconvenience and ease of understanding. However, the processesillustrated in FIGS. 7 and 8 are not limited to being performed usingthe system 600. Moreover, the system 600 described herein is not limitedto performing the processes illustrated in FIGS. 7 and 8 .

The processes 700 and 800 are illustrated as collections of blocks inlogical flow graphs, which represent sequences of operations that can beimplemented in hardware, software, or a combination thereof. In thecontext of software, the blocks represent computer-executableinstructions stored on one or more computer-readable storage media that,when executed by processor(s), perform the recited operations.Generally, computer-executable instructions include routines, programs,objects, components, data structures, and the like that performparticular functions or implement particular abstract data types. Theorder in which the operations are described is not intended to beconstrued as a limitation, and any number of the described blocks can becombined in any order and/or in parallel to implement the processes. Insome embodiments, one or more blocks of the process can be omittedentirely. Moreover, the processes 700 and 800 can be combined in wholeor in part with each other or with other processes.

FIG. 7 illustrates an example process 700 for training a model forpredicting whether a vehicle is going to enter a lane region in front ofanother vehicle (e.g., perform a cut-in maneuver), as described herein.

Block 702 illustrates receiving sensor data and/or log data associatedwith the sensor data. As described above, individual vehicles, such asthe vehicle 602, can include sensor component(s) 606. In at least oneexample, the sensor component(s) 606 can include lidar sensors, radarsensors, ultrasonic transducers, sonar sensors, location sensors (e.g.,GPS, compass, etc.), inertial sensors (e.g., inertial measurement units,accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB,IR, intensity, depth, etc.), wheel encoders, audio sensors, environmentsensors (e.g., temperature sensors, humidity sensors, light sensors,pressure sensors, etc.), ToF sensors, etc. The sensor component(s) 606can provide input to the vehicle computing device(s) 604. In someexamples, the sensor component(s) 606 can preprocess at least some ofthe sensor data prior to sending the sensor data to the vehiclecomputing device(s) 604. In at least one example, the sensorcomponent(s) 606 can send sensor data, via the network(s) 632, to thecomputing device(s) 634 at a particular frequency, after a lapse of apredetermined period of time, in near real-time, etc.

In at least one example, a plurality of vehicles (e.g., a fleet ofvehicles) can send sensor data, via the network(s) 632, to the computingdevice(s) 634. In some examples, such sensor data can be associated withdriving logs (i.e., “log data”), which can indicate how individualvehicles of the plurality of vehicles moved within respectiveenvironments over time. That is, such driving logs can be historicaldriving logs indicating the movement of individual vehicles of theplurality of vehicles over time. In some examples, such driving logs cancomprise sensor data—which can include output(s) of individual sensor(s)at a particular instance of time—that is received over time. In someexamples, such driving logs can comprise outputs based on the sensordata (e.g., processed sensor data) associated with a period of time.That is, in some examples, log data can include raw sensor data receivedfrom the sensor component(s) 606, as well as one or more downstreamoutputs (e.g., perception outputs, prediction outputs, planner outputs,control outputs, and the like) for individual messages during a drivemission of a vehicle, such as the vehicle 602.

In at least one example, the training component 640 of the computingdevice(s) 634 can receive sensor data and/or associated driving logsfrom the plurality of vehicles.

Block 704 illustrates mining the sensor data and/or the log data todetect events associated with cut-in maneuvers (e.g., any one or more ofthe maneuvers described in detail herein (whether or not the cut-inmaneuver is relative to a second vehicle)), wherein the detected eventsare annotated in the sensor data and/or log data to generate annotateddata. In at least one example, the training component 640 can mine(e.g., analyze) the sensor data and/or the log data to detect when avehicle performs a cut-in maneuver (e.g., which can be associated with aleft turn (e.g., from a parking spot (e.g., substantially perpendicularparking spot) or parking lot, an alley, a driveway, or the like), aright turn (e.g., from a parking spot (e.g., substantially perpendicularparking spot) or parking lot, an alley, a driveway, or the like), aU-turn, a K-turn, an N-point turn, or the like). For example, thetraining component 640 can analyze the sensor data and/or the log datato detect when a vehicle is misaligned with a driving surface, forinstance, when the vehicle is substantially perpendicular to a drivingsurface, and subsequently becomes aligned with, or substantiallyparallel to, the driving surface. In at least one example, theannotation component 642 can automatically annotate the sensor dataand/or the log data to indicate the occurrence of cut-in maneuvers.

In at least one example, the annotation component 642 can determine aperiod of time associated with a cut-in maneuver. That is, theannotation component 642 can access sensor data and/or log data leadingup to a detected event (e.g., associated with a cut-in maneuver), andcan determine when the detected event started and ended. For a movingvehicle (e.g., a vehicle making a u-turn, k-point turn, n-point turn,etc.), the annotation component 642 can backout (e.g., in the log data)to a first time that the vehicle starts to diverge from a driving laneto determine a start time associated with the event. For instance, theannotation component 642 can determine that a position of the vehiclechanges from being substantially parallel to a centerline of a drivingsurface (e.g., a road, a lane, etc.) to being within a designated offsetof 90 degrees of the centerline (e.g., substantially perpendicular tothe centerline) and can determine the start time associated with theevent accordingly. For a stopped vehicle (e.g., parked, idling in adriveway or alley, etc.), the annotation component 642 can utilizechanges in velocity to determine when the vehicle starts to perform acut-in maneuver. For example, the annotation component 642 can determinea first time when the vehicle moves from a zero velocity to a non-zerovelocity to determine a start time associated with the event. In atleast one example, the end time can correspond to when the event occurs.

Block 706 illustrates generating training data. In at least one example,the training component 640 can access sensor data and/or log dataleading up to a detected event (e.g., associated with a cut-inmaneuver), and can generate multi-channel image(s) associated with suchsensor data and/or log data. In at least one example, the annotationcomponent 642 can determine when the event associated with the cut-inmaneuver started and ended, as described above, and can access sensordata and/or log data from the start time to just prior to the occurrenceof the event. For instance, the training component 640 can access 1-10seconds of sensor data and/or log data prior to a detected event, whichcan be used to generate multi-channel image(s) representing the state ofthe environment and/or the vehicle prior to the occurrence of thedetected event.

In at least one example, the sensor data and/or log data can be encodedinto a multi-channel image or images. Each channel can correspond to afeature which for a vehicle that is detected as performing a cut-inmaneuver, can include, but is not limited to, annotations in map(s)associated with environment(s) indicating parking spot(s) (e.g.,substantially perpendicular parking spot(s)) or parking lot(s),alley(s), driveway(s), and the like), positions of a vehicle (e.g.,direction of travel, angular offset from another vehicle, etc.), aninstantaneous velocity of the vehicle, an indication of whether a driveris in the vehicle, an indication of a direction that the driver islooking (e.g., head direction of the driver), an indication of whether adoor associated with the vehicle is open or closed, an indication ofwhether an engine of the vehicle is in a running state or off state, awheel angle associated with wheel(s) of the vehicle, an indication ofwhether a brake light of the vehicle is illuminated, an indication ofwhether a headlight of the vehicle is illuminated, an indication ofwhether a reverse light of the vehicle is illuminated, an indication ofwhether a blinker of the vehicle is illuminated, or a lighting stateassociated with the vehicle, etc. In at least one example, each channelcan be associated with a top-down view of the environment. That is, achannel can represent sensor output or information based on a sensoroutput that is modeled in a top-down representation. The channel(s) canthen be input into a model, such as a multi-layer model, a convolutionalneural network, or the like. Additional details associated with top-downconvolutional neural networks are described in U.S. patent applicationSer. No. 16/420,050, which was filed on May 22, 2019, and U.S. patentapplication Ser. No. 16/504,147, which was filed on Jul. 5, 2019, theentire contents of both of which are incorporated by reference herein.Of course, though described above with respect to a multi-channel image,any number of multi-channel images (which may represent a series of timeprior to the most recent time step) and/or any other data structure arecontemplated.

In some examples, the training data, and associated sensor data and/orlog data, can be referred to as ground truth.

Block 708 illustrates inputting the training data into a model todetermine an output associated with an indication whether a vehicle islikely to perform a cut-in maneuver. In at least one example, thetraining data can be analyzed by the training component 640 using amachine learning mechanism. For instance, in at least one example, amodel can be trained using a neural network (e.g., a convolutionalneural network, etc.) and the training data. That is, the training datacan be input into a model (e.g., a top-down convolutional neuralnetwork, etc.) to determine an output associated with a prediction ofwhether a vehicle will enter a lane in front of another vehicle orotherwise perform a maneuver described herein as a “cut-in maneuver.”Such a “prediction” can be a detection of a cut-in maneuver, aprediction of a future lane or position in the environment, or somethingin between. In some examples, the output can be a binary indicator(e.g., “yes” or “no”), indicating whether the vehicle is going to enterthe driving lane of the other vehicle in front of the other vehicle ornot. In some examples, the output can be associated with a confidencescore. In some examples, the output can be a percentage or other metricindicating a certainty of whether a vehicle is going to enter thedriving lane of another vehicle in front of the other vehicle. Asdescribed above, in some examples, the output can be associated with apredicted lane or position in the environment.

Block 710 illustrates determining a difference between the output and anexpected output associated with the training data. In at least oneexample, the resulting model can determine an output associated with aprediction whether a vehicle will enter a lane in front of anothervehicle or otherwise perform a cut-in maneuver, as described herein. Thetraining component 640 can compare the output with an expected outputassociated with the training data, which can be based at least in parton the annotated data, to determine a difference. In at least oneexample, the training component 640 can modify one or more parameters ofthe model based at least in part on the difference (e.g., via gradientdescent, back-propagation, and the like).

Block 712 illustrates determining whether the difference is greater thanor equal to a threshold. In at least one example, the training component640 can determine whether the difference is greater than or equal to athreshold. If the difference is greater than or equal to the threshold,the training component 640 can modify a parameter of the model, asillustrated in block 714, and the process 700 can return to block 702.If the difference is less than the threshold, the training component 640can transmit the model to the model(s) storage 648 and/or storage on avehicle, such as the storage 630 on the vehicle 602. In some examples,and as illustrated in block 716, the model can be transmitted to avehicle, such as the vehicle 602, that can be configured to becontrolled by output(s) of the model. In such an example, the resultingmodel can be stored in the storage 630 on the vehicle 602 and can beaccessed in near real-time by one or more components of the vehiclecomputing device(s) 604, as described below with reference to FIG. 8 .

In at least one example, the training component 640 can train model(s)using machine learning techniques. Block 708 makes reference to one ormore neural networks that can be used to train the model. Additional oralternative machine learning algorithms can be used for training themodel. Such machine learning algorithms can include, but are not limitedto, regression algorithms (e.g., ordinary least squares regression(OLSR), linear regression, logistic regression, stepwise regression,multivariate adaptive regression splines (MARS), locally estimatedscatterplot smoothing (LOESS)), example-based algorithms (e.g., ridgeregression, least absolute shrinkage and selection operator (LASSO),elastic net, least-angle regression (LARS)), decisions tree algorithms(e.g., classification and regression tree (CART), iterative dichotomiser3 (ID3), Chi-squared automatic interaction detection (CHAID), decisionstump, conditional decision trees), Bayesian algorithms (e.g., naïveBayes, Gaussian naïve Bayes, multinomial naïve Bayes, averageone-dependence estimators (AODE), Bayesian belief network (BNN),Bayesian networks), clustering algorithms (e.g., k-means, k-medians,expectation maximization (EM), hierarchical clustering), associationrule learning algorithms (e.g., perceptron, back-propagation, hopfieldnetwork, Radial Basis Function Network (RBFN)), deep learning algorithms(e.g., Deep Boltzmann Machine (DBM), other Deep Belief Networks (DBN),Artificial Neural Network (ANN), Stacked Auto-Encoders), DimensionalityReduction Algorithms (e.g., Principal Component Analysis (PCA),Principal Component Regression (PCR), Partial Least Squares Regression(PLSR), Sammon Mapping, Multidimensional Scaling (MDS), ProjectionPursuit, Linear Discriminant Analysis (LDA), Mixture DiscriminantAnalysis (MDA), Quadratic Discriminant Analysis (QDA), FlexibleDiscriminant Analysis (FDA)), SVM (support vector machine), supervisedlearning, unsupervised learning, semi-supervised learning, etc.

FIG. 8 illustrates an example process 800 for predicting whether avehicle is going to enter a lane region in front of another vehicle, andcontrolling the other vehicle based on such a prediction, as describedherein.

Block 802 illustrates receiving sensor data associated with anenvironment of a vehicle. As described above, individual vehicles, suchas the vehicle 602, can include sensor component(s) 606. In at least oneexample, the sensor component(s) 606 can include lidar sensors, radarsensors, ultrasonic transducers, sonar sensors, location sensors (e.g.,GPS, compass, etc.), inertial sensors (e.g., inertial measurement units,accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB,IR, intensity, depth, etc.), wheel encoders, audio sensors, environmentsensors (e.g., temperature sensors, humidity sensors, light sensors,pressure sensors, etc.), ToF sensors, etc. The sensor component(s) 606can provide input to the vehicle computing device(s) 604.

Block 804 illustrates determining whether another vehicle is detectedproximate to the vehicle. As described above, the vehicle 602 caninclude one or more components associated with the vehicle computingdevice(s) 604 onboard the vehicle 602. In at least one example, thelocalization component 620 can determine a pose (position andorientation) of the vehicle 602 in relation to a local and/or global mapbased at least in part on sensor data received from the sensorcomponent(s) 606 and/or map data associated with a map (e.g., of themap(s)). In at least one example, the localization component 620 caninclude, or be associated with, a calibration component that is capableof performing operations for calibrating (determining various intrinsicand extrinsic parameters associated with any one or more of the sensorcomponent(s) 606), localizing, and mapping substantially simultaneously.

In at least one example, the perception component 622 can perform objectdetection, segmentation, and/or classification based at least in part onsensor data received from the sensor component(s) 606. In at least oneexample, the perception component 622 can receive raw sensor data (e.g.,from the sensor component(s) 606). In at least one example, theperception component 622 can receive sensor data and can utilize one ormore processing algorithms to perform object detection, segmentation,and/or classification with respect to object(s) identified in the sensordata. In some examples, the perception component 622 can associate abounding region (or otherwise an instance segmentation) with anidentified object and can associate a confidence score associated with aclassification of the identified object with the identified object. Insome examples, objects, when rendered via a display, can be coloredbased on their perceived class.

In at least one example, the prediction component 624 can analyzelocalization data (e.g., output from the localization component 620) andperception data (e.g., output from the perception component 622) todetermine whether the perception data indicates that another vehicle iswithin a threshold distance of the vehicle 602. That is, the predictioncomponent 624 can determine whether another vehicle is proximate thevehicle 602. If no vehicle is determined to be proximate the vehicle602, the process 800 can return to block 802. If another vehicle isdetermined to be proximate the vehicle 602, the prediction component 624can determine an output indicate whether the other vehicle is predictedto enter a lane in front of the vehicle 602, as described below.

Block 806 illustrates determining whether the other vehicle is predictedto perform a cut-in maneuver. In at least one example, the predictioncomponent 624 can use the model, trained via machine learning techniquesdescribed above with reference to FIG. 7 , to output a predictionindicating whether the other vehicle is going to enter the driving laneof the vehicle 602 in front of the vehicle 602, or otherwise perform acut-in maneuver. That is, the prediction component 624 can use a modelto generate a prediction regarding whether the other vehicle is likelyto perform a cut-in maneuver (e.g., enter a driving lane in front of thevehicle 602).

In at least one example, the prediction component 624 can convert thesensor data (e.g., received in block 802) into a top-down representationof the environment. For example, the prediction component 624 cangenerate multi-channel image(s) associated with such sensor data. Asdescribed above, each channel can correspond to a feature which caninclude, but is not limited to, annotations in map(s) associated withthe environment indicating parking spot(s) (e.g., substantiallyperpendicular parking spot(s)) or parking lot(s), alley(s), driveway(s),and the like), positions of the vehicle and/or the other vehicle (e.g.,direction of travel, angular offset from another vehicle, etc.), aninstantaneous velocity of the other vehicle, an indication of whether adriver is in the other vehicle, an indication of a direction that thedriver is looking (e.g., head direction of the driver), an indication ofwhether a door associated with the other vehicle is open or closed, anindication of whether an engine of the other vehicle is in a runningstate or off state, a wheel angle associated with wheel(s) of the othervehicle, an indication of whether a brake light of the other vehicle isilluminated, an indication of whether a headlight of the other vehicleis illuminated, an indication of whether a reverse light of the othervehicle is illuminated, an indication of whether a blinker of the othervehicle is illuminated, or a lighting state associated with the othervehicle, etc. In at least one example, each channel can be associatedwith a top-down view of the environment. That is, a channel canrepresent sensor output or information based on a sensor output that ismodeled in a top-down representation. As above, multiple multi-channelimages representing a period of time preceding a most recent time step(or any other data structure) are contemplated.

In at least one example, the model can be trained to analyze one or morefeatures (also referred to as attributes, characteristics, parameters,data, and the like) associated with the environment within which thesensor data is associated and/or the other vehicle to output aprediction with respect to whether the other vehicle is going to enterthe driving lane of the vehicle 602 in front of the vehicle 602. In atleast one example, the prediction component 624 can receive an output(e.g., from the respective model) that indicates a likelihood that theother vehicle is going to enter the driving lane of the vehicle 602 infront of the vehicle 602. As described above, in some examples, theoutput can be a binary indicator (e.g., “yes” or “no”), indicatingwhether the other vehicle is going to enter the driving lane of thevehicle 602 in front of the vehicle 602 or not. In some examples, theoutput can be a percentage or other metric indicating a certainty ofwhether the other vehicle is going to enter the driving lane of thevehicle 602 in front of the vehicle 602. In such examples, theprediction component 624 can compare the output percentage or othermetric to a threshold and, if the output percentage or other metric isgreater than or equal to the threshold, the prediction component 624 candetermine that the other vehicle is predicted to enter into the drivinglane of the vehicle 602 in front of the vehicle. If the outputpercentage or other metric is less than the threshold, the predictioncomponent 624 can determine that the other vehicle is predicted to enterinto the driving lane of the vehicle 602 in front of the vehicle

In at least one example, if the other vehicle is predicted to enter intoa lane in front of the vehicle 602 (e.g., perform a cut-in maneuver),the planning component 626 associated with the vehicle 602 can receivean indication of such. The planning component 626 can determine a firstinstruction for controlling the vehicle 602 in view of the othervehicle, as illustrated in block 808, and can cause the vehicle 602 tobe controlled based at least in part on the first instruction, asillustrated in block 810. That is, as described above, the planningcomponent 626 can utilize the output of the model to determine how tonavigate the vehicle 602 in its environment. For example, the planningcomponent 626 can generate one or more trajectories that can be used byother components associated with the vehicle 602 to adapt drivingoperations based on predictions output by the model. In some examples,such trajectory(s) can be modifications of previous trajectory(s). Insome examples, such trajectory(s) can be new trajectory(s) based on thefirst instruction. In view of the foregoing, the planning component 626can determine a trajectory to navigate the vehicle 602 to accommodatethe other vehicle, for example, by causing the vehicle 602 to deceleratethereby increasing a follow distance between the vehicle 602 and anothervehicle, causing the vehicle 602 to decelerate to yield to the othervehicle and/or stop, causing the vehicle 602 to perform a lane change,causing the vehicle 602 to safely maneuver around the other vehicle,and/or any other combination of maneuvers the vehicle 602 is capable ofperforming, etc.

In some examples, the output can be used by the prediction component 624to predict how the other vehicle is likely to maneuver and theprediction component 624 can output a trajectory representative of sucha prediction. In such examples, the predicted trajectory associated withthe other vehicle can be provided to the planning component 626 for theplanning component to determine the trajectory for navigating thevehicle 602.

In at least one example, if the other vehicle is not predicted to enterinto a lane in front of the vehicle 602, the planning component 626associated with the vehicle 602 can receive an indication of such. Insuch an example, the planning component 626 can determine a secondinstruction for controlling the vehicle 602, as illustrated in block812, and can cause the vehicle 602 to be controlled based at least inpart on the second instruction, as illustrated in block 814. In such anexample, if the other vehicle is not predicted to enter into the lane infront of the vehicle 602, the planning component 626 can plan atrajectory that otherwise controls the vehicle 602 to travel within theenvironment (which may or may not take the other car intoconsideration). In some examples, such a trajectory can be the sametrajectory as previously determined (e.g., prior to determining that theother vehicle is not predicted to perform a cut-in maneuver). In someexamples, the trajectory can be a new trajectory based on the secondinstruction.

FIGS. 1-5 , described above, illustrate various examples of othervehicles that are proximate to a vehicle, such as the vehicle 602, thatare entering a lane region of the driving lane in front of the vehicle602 or are otherwise performing a “cut-in maneuver,” as describedherein. Such examples include left turns, right turns, U-turns,three-point turns, and the like. As described above, additional oralternative examples are within the scope of this disclosure (e.g.,K-turns, any N-point turn, etc.). That is, the process 800, describedabove, can be applicable to any example where another vehicle that isproximate to and associated with a same or different direction of travelthan the vehicle 602 enters—or is predicted to enter—a lane region ofthe driving lane in front of the vehicle.

Example Clauses

A. A method comprising: receiving log data associated with vehicles inan environment; detecting an event in the log data, wherein an eventcorresponds to a cut-in maneuver performed by a vehicle; determining aperiod of time associated with the event; accessing a portion of the logdata that corresponds to the period of time; generating training databased at least in part on converting the portion of the log data thatcorresponds to the period of time into a top-down representation of theenvironment; inputting the training data into a model; determining, bythe model, an output comprising an indication that another vehicle islikely to perform another cut-in maneuver; determining a differencebetween the output and an expected output associated with the trainingdata; altering one or more parameters of the model based on thedifference; and transmitting the model to an autonomous vehicleconfigured to be controlled by another output of the model.

B. The method as paragraph A recites, wherein the log data comprises rawsensor data and a downstream output determined based at least in part onthe raw sensor data, wherein the downstream output comprises perceptiondata, prediction data, or planner data.

C. The method as paragraph A or B recites, further comprising:determining that a position of the vehicle changes from beingsubstantially parallel to a centerline of a driving surface to within adesignated offset of 90 degrees of the centerline; and determining astart of the period of time based at least in part on determining whenthe position of the vehicle changes from being substantially parallel tothe centerline to within the designated offset of 90 degrees of thecenterline.

D. The method as any of paragraphs A-C recites, further comprising:determining that a velocity of the vehicle changes from zero to anon-zero velocity prior to the event; and determining a start of theperiod of time based at least in part on determining when the velocityof the vehicle changes from zero to the non-zero velocity prior to theevent.

E. A system comprising: one or more processors; and one or morenon-transitory computer-readable media storing instructions, that whenexecuted by the one or more processors, cause the system to performoperations comprising: receiving log data associated with vehicles in anenvironment; detecting an event in the log data, wherein an eventcorresponds to a cut-in maneuver performed by a vehicle; generatingtraining data based at least in part on converting a portion of the logdata that corresponds to the event into a top-down representation of theenvironment; inputting the training data into a model; and transmittingthe model to an autonomous vehicle configured to be controlled byanother output of the model.

F. The system as paragraph E recites, the operations further comprising:determining a period of time associated with the event; and accessing aportion of the log data that corresponds to the period of time; whereinthe top-down representation of the environment is associated with theportion of the log data that corresponds to the period of time.

G. The system as paragraph F recites, the operations further comprising:determining that a position of the vehicle changes from beingsubstantially parallel to a centerline of a driving surface to within adesignated offset of 90 degrees of the centerline; and determining astart of the period of time based at least in part on determining whenthe position of the vehicle changes from being substantially parallel tothe centerline to within the designated offset of 90 degrees of thecenterline.

H. The system as paragraph F or G recites, the operations furthercomprising: determining that a velocity of the vehicle changes from zeroto a non-zero velocity prior to the event; and determining a start ofthe period of time based at least in part on determining when thevelocity of the vehicle changes from zero to the non-zero velocity priorto the event.

I. The system as any of paragraphs E-H recites, the operations furthercomprising: determining, by the model, an output comprising anindication that another vehicle is likely to perform another cut-inmaneuver; determining a difference between the output and an expectedoutput associated with the training data; and altering one or moreparameters of the model based on the difference.

J. The system as any of paragraphs E-I recites, wherein the modelcomprises a convolutional neural network.

K. The system as any of paragraphs E-J recites, wherein the top-downrepresentation comprises a multi-channel image, wherein each channelcorresponds to at least one of: a position of the vehicle relative to acenterline of a driving surface; an instantaneous velocity of thevehicle; an indication of whether a driver is in the vehicle; anindication of whether a door associated with the vehicle is open orclosed; an indication of whether an engine of the vehicle is in arunning state or off state; an indication of whether a brake light ofthe vehicle is illuminated; an indication of whether a headlight of thevehicle is illuminated; an indication of whether a reverse light of thevehicle is illuminated; or an indication of whether a blinker of thevehicle is illuminated.

L. The system as any of paragraphs E-K recites, wherein the cut-inmaneuver comprises any one of: a left turn from an alley, a parkingspot, or a driveway; a right turn from an alley, a parking spot, or adriveway; a u-turn; an n-point turn; a k-turn; or a backward reverseinto a driving lane.

M. One or more non-transitory computer-readable media storinginstructions, that when executed by one or more processors, cause theone or more processors to perform operations comprising: receiving logdata associated with vehicles in an environment; detecting an event inthe log data, wherein an event corresponds to a cut-in maneuverperformed by a vehicle; generating training data based at least in parton converting a portion of the log data that corresponds to the eventinto a top-down representation of the environment; and inputting thetraining data into a model, wherein the model is trained to output anindication of whether another vehicle is likely to perform anothercut-in maneuver.

N. The one or more non-transitory computer-readable media as paragraph Mrecites, the operations further comprising transmitting the model to anautonomous vehicle configured to be controlled by another output of themodel.

O. The one or more non-transitory computer-readable media as paragraph Mor N recites, the operations further comprising: determining a period oftime associated with the event; accessing a portion of the log data thatcorresponds to the period of time; wherein the top-down representationof the environment is associated with the portion of the log data thatcorresponds to the period of time.

P. The one or more non-transitory computer-readable media as paragraph Orecites, wherein determining the period of time is based on at least oneof a change in a position of the vehicle or a velocity of the vehicle.

Q. The one or more non-transitory computer-readable media as any ofparagraphs M-P recites, the operations further comprising: determining adifference between an output of the model and an expected outputassociated with the training data; and altering one or more parametersof the model based on the difference.

R. The one or more non-transitory computer-readable media as any ofparagraphs M-Q recites, wherein the model comprises a convolutionalneural network.

S. The one or more non-transitory computer-readable media as any ofparagraphs M-R recites, wherein the top-down representation comprises amulti-channel image, wherein each channel corresponds to at least oneof: a position of the vehicle relative to a centerline of a drivingsurface; an instantaneous velocity of the vehicle; an indication ofwhether a driver is in the vehicle; an indication of whether a doorassociated with the vehicle is open or closed; an indication of whetheran engine of the vehicle is in a running state or off state; anindication of whether a brake light of the vehicle is illuminated; anindication of whether a headlight of the vehicle is illuminated; anindication of whether a reverse light of the vehicle is illuminated; oran indication of whether a blinker of the vehicle is illuminated.

T. The one or more non-transitory computer-readable media as any ofparagraphs M-S recites, wherein the cut-in maneuver comprises any oneof: a left turn from an alley, a parking spot, or a driveway; a rightturn from an alley, a parking spot, or a driveway; a u-turn; an n-pointturn; a k-turn; or a backward reverse into a driving lane.

U. A method comprising: receiving sensor data associated with anenvironment of a first vehicle; detecting, based at least in part on thesensor data, a second vehicle proximate the first vehicle, wherein thesecond vehicle is substantially perpendicular to the first vehicle;determining, based at least in part on the sensor data, an attributeassociated with the environment or the second vehicle that is indicativeof whether the second vehicle is going to enter a lane region in frontof the first vehicle; determining, based at least in part on theattribute, an indication whether the second vehicle will enter the laneregion in front of the first vehicle; and determining an instruction forcontrolling the first vehicle based at least in part on the indication.

V. The method as paragraph U recites, wherein the second vehicle issubstantially perpendicular to the first vehicle based at least in parton an angle between the second vehicle and the first vehicle beingwithin a designated offset of 90 degrees.

W. The method as paragraph V recites, wherein the designated offset is50 degrees.

X. The method as any of paragraphs U-W recites, wherein the secondvehicle is positioned in a parking spot, driveway, or alley that issubstantially perpendicular to the first vehicle.

Y. The method as any of paragraphs U-X recites, wherein the secondvehicle is associated with a different direction of travel than thefirst vehicle.

Z. The method as any of paragraphs U-Y recites, wherein the attributecomprises at least one of: a position of the second vehicle relative tothe first vehicle; an instantaneous velocity of the second vehicle; anindication of whether a driver is in the second vehicle; an indicationof whether a door associated with the second vehicle is open or closed;an indication of whether an engine of the second vehicle is in a runningstate or off state; an indication of whether a brake light of the secondvehicle is illuminated; an indication of whether a headlight of thesecond vehicle is illuminated; an indication of whether a reverse lightof the second vehicle is illuminated; or an indication of whether ablinker of the second vehicle is illuminated.

AA. A system comprising: one or more processors; and one or morenon-transitory computer-readable media storing instructions, that whenexecuted by the one or more processors, cause the system to performoperations comprising: receiving sensor data associated with anenvironment of a first vehicle; determining, based at least in part onthe sensor data, that a second vehicle proximate the first vehicle ispredicted to enter a lane region in front of the first vehicle; anddetermining an instruction for controlling the first vehicle based atleast in part on determining that the second vehicle proximate the firstvehicle is predicted to enter the lane region in front of the firstvehicle.

AB. The system as paragraph AA recites, further comprising determiningthat the second vehicle is predicted to enter the lane region in frontof the first vehicle is based at least in part on determining one ormore attributes associated with at least one of the second vehicle orthe environment.

AC. The system as paragraph AB recites, wherein the one or moreattributes comprise at least one of: a position of the second vehiclerelative to the first vehicle; an instantaneous velocity of the secondvehicle; an indication of whether a driver is in the second vehicle; anindication of whether a door associated with the second vehicle is openor closed; an indication of whether an engine of the second vehicle isin a running state or off state; an indication of whether a brake lightof the second vehicle is illuminated; an indication of whether aheadlight of the second vehicle is illuminated; an indication of whethera reverse light of the second vehicle is illuminated; or an indicationof whether a blinker of the second vehicle is illuminated.

AD. The system as any of paragraphs AA-AC recites, wherein the firstvehicle and the second vehicle are positioned at an angle between adesignated offset from 90 degrees.

AE. The system as paragraph AD recites, wherein the second vehicle ispositioned in a parking spot or a driveway on a same side of a road asthe first vehicle or on an opposite side of a road as the first vehicle.

AF. The system as any of paragraphs AA-AE recites, wherein determiningan instruction for controlling the first vehicle is based at least inpart on an output from a machine-trained model, wherein the outputcomprises a binary indication whether the second vehicle will enter thelane region in front of the first vehicle or a percentage indicating acertainty associated with whether the second vehicle will enter the laneregion in front of the first vehicle.

AG. The system as any of paragraphs AA-AF recites, wherein theinstruction causes the first vehicle to at least one of decelerate,stop, or perform a lane change operation.

AH. One or more non-transitory computer-readable media storinginstructions, that when executed by one or more processors, cause theone or more processors to perform operations comprising: receivingsensor data associated with an environment of a first vehicle;converting the sensor data into a top-down representation of theenvironment; inputting the top-down representation into amachine-trained model; determining, based at least in part on inputtingthe top-down representation into the machine-trained model, that asecond vehicle proximate the first vehicle is predicted to enter a laneregion in front of the first vehicle; and determining an instruction forcontrolling the first vehicle based at least in part on determining thatthe second vehicle proximate the first vehicle is predicted to enter thelane region in front of the first vehicle.

AI. The one or more non-transitory computer-readable media as paragraphAH recites, wherein determining that the second vehicle proximate thefirst vehicle is predicted to enter the lane region in front of thefirst vehicle is based at least in part on at least one attribute,wherein the attribute comprises: an instantaneous velocity of the secondvehicle; an indication of whether a driver is in the second vehicle and,if the driver is in the second vehicle, an indication of a headdirection of the driver; an indication of whether an engine of thesecond vehicle is in a running state or an off state; a wheel angleassociated with a wheel of the second vehicle; or an indication ofwhether at least one of a brake light, a headlight, a reverse light, ora blinker of the second vehicle is illuminated.

AJ. The one or more non-transitory computer-readable media as paragraphAH or AI recites, wherein the second vehicle is associated with adifferent direction of travel than the first vehicle.

AK. The one or more non-transitory computer-readable media as any ofparagraphs AH-AJ recites, wherein the second vehicle and the firstvehicle are positioned at an angle within a designated offset of 90degrees.

AL. The one or more non-transitory computer-readable media as any ofparagraphs AH-AK recites wherein the second vehicle is positioned ineither (i) a parking spot or a driveway on a same side of a road as thefirst vehicle or (ii) a parking spot or a driveway on an opposite sideof a road as the first vehicle.

AM. The one or more non-transitory computer-readable media as any ofparagraphs AH-AL recites, the operations further comprising: inputting,into a prediction component, an indication that a second vehicleproximate the first vehicle is predicted to enter a lane region in frontof the first vehicle; receiving, from the prediction component, apredicted trajectory associated with the second vehicle; and determiningthe instruction further based at least in part on the predictedtrajectory.

AN. The one or more non-transitory computer-readable media as any ofparagraphs AH-AM recites, wherein the instruction causes the firstvehicle to at least one of decelerate or perform a lane changeoperation.

While the example clauses described above are described with respect toone particular implementation, it should be understood that, in thecontext of this document, the content of the example clauses can also beimplemented via a method, device, system, a computer-readable medium,and/or another implementation. Additionally, any of examples A-AN may beimplemented alone or in combination with any other one or more of theexamples A-AN.

CONCLUSION

While one or more examples of the techniques described herein have beendescribed, various alterations, additions, permutations and equivalentsthereof are included within the scope of the techniques describedherein.

In the description of examples, reference is made to the accompanyingdrawings that form a part hereof, which show by way of illustrationspecific examples of the claimed subject matter. It is to be understoodthat other examples can be used and that changes or alterations, such asstructural changes, can be made. Such examples, changes or alterationsare not necessarily departures from the scope with respect to theintended claimed subject matter. While the steps herein can be presentedin a certain order, in some cases the ordering can be changed so thatcertain inputs are provided at different times or in a different orderwithout changing the function of the systems and methods described. Thedisclosed procedures could also be executed in different orders.Additionally, various computations that are herein need not be performedin the order disclosed, and other examples using alternative orderingsof the computations could be readily implemented. In addition to beingreordered, the computations could also be decomposed intosub-computations with the same results.

1.-20. (canceled)
 21. A method comprising: receiving sensor data associated with an environment of a first vehicle; detecting, based at least in part on the sensor data, a second vehicle proximate the first vehicle, wherein the second vehicle is in an unstructured region and is positioned between a first angle and a second angle relative to the first vehicle, the unstructured region including unstructured portions of map data; determining, based at least in part on the sensor data, an attribute associated with at least one of the environment or the second vehicle that is indicative of whether the second vehicle that begins from the unstructured region is going to enter a lane region associated with the first vehicle; determining, based at least in part on the attribute, an indication of whether the second vehicle will enter the lane region associated with the first vehicle, wherein determining the indication is performed, at least in part, by a machine learned model trained to predict behavior of objects in environments; determining an instruction for controlling the first vehicle based at least in part on the indication; and controlling the first vehicle based at least in part on the instruction.
 22. The method of claim 21, wherein the second vehicle is positioned in at least one of a parking spot, driveway, or alley that is substantially perpendicular to the first vehicle.
 23. The method of claim 21, wherein the machine learned model is trained to predict the behavior of objects based on sensor data and unstructured portions of map data.
 24. The method of claim 21, wherein the attribute comprises at least one of: a position of the second vehicle relative to the first vehicle; an instantaneous velocity of the second vehicle; an indication of whether a driver is in the second vehicle; an indication of whether a door associated with the second vehicle is open or closed; an indication of whether an engine of the second vehicle is in a running state or off state; an indication of whether a brake light of the second vehicle is illuminated; an indication of whether a headlight of the second vehicle is illuminated; an indication of whether a reverse light of the second vehicle is illuminated; or an indication of whether a blinker of the second vehicle is illuminated.
 25. The method of claim 21, wherein the unstructured region represents at least one of a driveway, parking lot, or region in which there are no lane markers.
 26. The method of claim 21, wherein the second vehicle transitions from an unstructured portion of map data to a structured portion of map data.
 27. The method of claim 21, wherein the machine learned model is trained using multi-channel images, wherein a respective channel of the multi-channel images includes map data.
 28. A system comprising: one or more processors; and one or more non-transitory computer-readable media storing instructions, that when executed by the one or more processors, cause the system to perform operations comprising: receiving sensor data associated with an environment of a first vehicle; determining that a second vehicle proximate the first vehicle is predicted to enter a lane region in front of the first vehicle, wherein the second vehicle is in an unstructured region and is positioned between a first angle and a second angle relative to the first vehicle, the unstructured region including unstructured portions of map data, wherein determining that the second vehicle that begins from the unstructured region will enter the lane region in front of the first vehicle is performed, at least in part, by a machine learned model trained to predict behavior of objects in an environment; determining an instruction for controlling the first vehicle based at least in part on determining that the second vehicle proximate the first vehicle is predicted to enter the lane region in front of the first vehicle; and controlling the first vehicle, based at least in part on the instruction.
 29. The system of claim 28, further comprising determining that the second vehicle is predicted to enter the lane region in front of the first vehicle is based at least in part on determining one or more attributes associated with at least one of the second vehicle or the environment.
 30. The system of claim 29, wherein the one or more attributes comprise at least one of: a position of the second vehicle relative to the first vehicle; an instantaneous velocity of the second vehicle; an indication of whether a driver is in the second vehicle; an indication of whether a door associated with the second vehicle is open or closed; an indication of whether an engine of the second vehicle is in a running state or off state; an indication of whether a brake light of the second vehicle is illuminated; an indication of whether a headlight of the second vehicle is illuminated; an indication of whether a reverse light of the second vehicle is illuminated; or an indication of whether a blinker of the second vehicle is illuminated.
 31. The system of claim 28, wherein the second vehicle is positioned in a parking spot or a driveway on a same side of a road as the first vehicle or on an opposite side of a road as the first vehicle.
 32. The system of claim 28, wherein determining an instruction for controlling the first vehicle is based at least in part on an output from a machine-trained model, wherein the output comprises a binary indication whether the second vehicle will enter the lane region in front of the first vehicle or a percentage indicating a certainty associated with whether the second vehicle will enter the lane region in front of the first vehicle.
 33. The system of claim 28, wherein the instruction causes the first vehicle to at least one of decelerate, stop, or perform a lane change operation.
 34. The system of claim 28, the operations further comprising: converting the sensor data into a top-down representation of the environment; and inputting the top-down representation into a machine-trained model.
 35. One or more non-transitory computer-readable media storing instructions, that when executed by one or more processors, cause the one or more processors to perform operations comprising: receiving sensor data associated with an environment of a first vehicle; converting the sensor data into a top-down representation of the environment; inputting the top-down representation into a machine-trained model; determining, based at least in part on inputting the top-down representation into the machine-trained model, an indication that a second vehicle in an unstructured region and proximate the first vehicle is predicted to enter a lane region in front of the first vehicle, the unstructured region including unstructured portions of map data, wherein determining that the second vehicle that begins from the unstructured region is predicted to enter the lane region in front of the first vehicle from the unstructured region is performed, at least in part, by a machine learned model trained to predict behavior of objects in an environment from sensor data and map data; and determining an instruction for controlling the first vehicle based at least in part on determining that the second vehicle proximate the first vehicle is predicted to enter the lane region in front of the first vehicle.
 36. The one or more non-transitory computer-readable media of claim 35, wherein determining that the second vehicle proximate the first vehicle is predicted to enter the lane region in front of the first vehicle is based at least in part on an attribute, wherein the attribute comprises: an instantaneous velocity of the second vehicle; an indication of whether a driver is in the second vehicle and, if the driver is in the second vehicle, an indication of a head direction of the driver; an indication of whether an engine of the second vehicle is in a running state or an off state; a wheel angle associated with a wheel of the second vehicle; or an indication of whether at least one of a brake light, a headlight, a reverse light, or a blinker of the second vehicle is illuminated.
 37. The one or more non-transitory computer-readable media of claim 35, wherein the second vehicle is associated with a different direction of travel than the first vehicle.
 38. The one or more non-transitory computer-readable media of claim 35, wherein the second vehicle and the first vehicle are positioned at an angle of between 40 degrees and 140 degrees.
 39. The one or more non-transitory computer-readable media of claim 35, wherein the second vehicle is positioned in either (i) a parking spot or a driveway on a same side of a road as the first vehicle or (ii) a parking spot or a driveway on an opposite side of a road as the first vehicle.
 40. The one or more non-transitory computer-readable media of claim 35, the operations further comprising: inputting, into a prediction component, the indication that the second vehicle in an unstructured region and proximate the first vehicle is predicted to enter the lane region in front of the first vehicle; receiving, from the prediction component, a predicted trajectory associated with the second vehicle; and determining the instruction further based at least in part on the predicted trajectory. 